The state of the art in combining real world imagery with additional imagery from another source is a process that requires careful control over which sections of each image are to be used in the final composite image. One common application is to combine images generated by a computer with images acquired from a traditional motion picture, video or digital camera. In order to seamlessly combine the images, the areas of each image that are to be preserved or modified must be defined. These areas are typically called mattes.
Mattes may be defined in a number of ways. In traditional compositing, the mattes are frequently defined by having an artist mark points around the perimeter of the object to be preserved or removed. The computer then connects the dots to form a closed shape, which forms the matte. Problems can arise, however, if the object and/or the camera move relative to the other.
In traditional computer compositing, a moving camera or object is handled by making a tracking matte, or a matte that moves along with the object. While the methods of moving the matte along with the object vary, they typically center around having the user specify an area of high contrast in the live action image, measuring how that image moves around in the frame, and connecting the motion of the drawn matte to the motion of the high contrast object.
This process works, but has several limitations. Firstly, if the high contrast area is located on the front of a character's shirt, for example, and the character turns around, or if the camera moves around to another side of the character, the local effect is destroyed. Secondly, the process of measuring the camera motion by tracking the individual pixels of the high contrast part of the image is both fragile and time-consuming if there is no additional camera data to work from. It typically cannot be computed in real time, and if a frame of the live action image has a lighting change where the pattern is unrecognizable, the artist must re-specify the high contrast area at the frame of failure to continue the process. The process of creating all of the multiple overlapping mattes that are used in a sophisticated visual effects shot can exceed the time required to complete the rest of the shot due to the handwork required.
In addition, if the live action camera is zoomed in, the high contrast area that was being tracked can simply disappear from the image, resulting in the matte failing to track the camera lens change.
Accordingly, the pixel tracking based methods do not work well for the demands of real time visual effects processing, which must be very rapid to compute as well as robust to the frame by frame changes in the live action video image.
In real time processing, mattes have traditionally been created by surveying the edges of the green screen background using an architectural measurement tool such as a total station, and creating a model of the matte in 3D space. However, models of this type cannot be rapidly modified by the artist under typical time pressure conditions found in entertainment production.